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ABSTRACT 

This paper presents a surveillance robot that performs face recognition using artificial intelligence (AT), 
controlled via an Android application for examining threats in its vicinity. The robot is capable of detecting humans in 
real time scenarios. The activity cycle of the robot is categorized in a number of tasks: Object Detection, Face Detection, 
Face Recognition, generation of alert signals to the user. The robot detects the human face based on Haar Cascade 
Algorithm using a camera module interfaced with Raspberry Pi. The system is exposed to numerous facial images of 
people from different angles and background conditions. This forms the database to train the system which aids in facial 
recognition. Subsequently, when the robot detects a human face, it initiates the algorithm for recognition of face which 
uses Local Binary Pattern (LBP) approach. According to images in the dataset, the robot determines whether the human 
detected is a threat or not. In case of a match with the dataset, the user receives a message through GSM module on 
his/her mobile phone along with the name of the person recognized. In case of a mismatch, a threat or an alert message 
is received by the user. The motion of the robot is controlled by the android application through Bluetooth. 
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INTRODUCTION 

Technological progress made closed-circuit television (CCTV) popular in office as well as home 
surveillance applications. Surveillance of each and every corner in the space using a CCTV is not cost effective. 
Also, this surveillance requires a human operator to detect and recognize the person and hence there are chances of 
human errors. CCTV systems alone cannot give 100% surveillance, hence, in this paper, we present a surveillance 
robot which has face detection and recognition capability also its movement can be controlled using an Android 
phone. The robot sends an alert message to the user that the detected human is a threat or not. The main advantage 
of the robot is that the user can recognize the human without even going to close vicinity. 

In this paper, the image is captured by the camera module (PICAMERA) and algorithms for face detection 
and face recognition are implemented in Raspberry Pi 3. Ada boost machine learning algorithm is used for face 
detection, which was proposed in the year 2001 by Paul Viola et al., This algorithm is based on Haar-like features 
which use a cascade of classifiers. This feature provides for human face detection irrespective of the background 
conditions, size, shape and color of image [2]. For fast and reliable face recognition process, a Linear Binary Pattern 
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algorithm is used. The digital image of a detected face is divided into pixels which are used for further processing. Features 
of this image are compared pixel by pixel with the features of faces in the database [4], [5]. An Arduino Uno hardware unit 
is controlling the activities such as motion detection using PIR sensor, which initiates the face recognition process and also 
controls the movement of the robot via an Android Phone. Upon the recognition of human, an alert message is generated 
which is broadcasted to a group of owners. 

The paper structure is as follows: section II presents the implementation and execution steps of the surveillance 
robot. Section III and IV discuss hardware and software specifications respectively in detail. Section V summarizes the 
experimental results obtained. Finally, section VI provides conclusion and puts forth future work. 

IMPLEMENTATION 

Flow structure of the activity cycle is described in Fig 1. Each surveillance cycle executed by the robot is divided 
into two states; the idle state and the active state. 

In the idle state, the robot stays static on the floor inside the office during the day and around the office compound 
at night. Meanwhile, the PIR sensor continuously searches for any object movements in its vicinity. The sun radiates a high 
amount of infrared radiations and hence the PIR sensor provides false alarm during the daytime. So, the PIR sensor is 
deployed inside office premise for protection during the day. If the robot senses the IR radiations from a particular 
direction, it enters the active state. Since the robot is static the power consumption in this state is in minimum. It is 
advantageous to deploy a static robot rather than an autonomous continuously moving robot in an area where the physical 
parameters of the place are unknown. 

In the active state, the Camera module is triggered using I2C communication between Arduino Uno and 
Raspberry Pi. Camera module is mounted at an angle of 45 degrees with respect to ground. It captures an image of the 
object and initiates the face detection process. If a human face is detected, the robot starts the face recognition algorithm. 
In case of a match with the dataset, the robot sends an SMS to the user along with the name of the person recognized, using 
GSM module. Otherwise, if the recognized face is of an intruder, immediately an alert SMS is sent to the user through 
GSM module. If a human face is not detected, the robot returns back to the idle state and continues searching for IR 
radiations. 

The robot always remains in the idle state when no person enters the premise. In both the states, the robot can be 
controlled by the Android application over Bluetooth. 
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Figure 1: Execution Flowchart 


HARDWARE SPECIFICATION 

Control Unit 

The Surveillance Robot is designed to have Raspberry Pi 3 (Model B) and Arduino Uno in its control unit. The 
advantage of having Raspberry pi over the other microcontrollers is that Raspberry Pi is a full-fledged credit card sized 
computer with 900 MHz microprocessor [7]. The onboard Wi-Fi module and Bluetooth Module makes it much better than 
Raspberry Pi 2(Model B+) and it is also equipped with a better RAM of 900MHz [7]. The Raspberry Pi also has Camera 
Interface and General-Purpose Input and Output (GPIO) pins for interfacing with other peripherals. The Universal 
Asynchronous Receiver Transmitter (UART) pins are interfaced with GSM Module. Since the Raspbian Operating System 
installed on Raspberry Pi is not a Real Time Operating System (RTOS) the motor control cannot be done using this 
controller. Hence Arduino Uno is used to serve the purpose. The Uno communicates with Raspberry Pi through I2C 
protocol. For the ease of operation of human detection, the PIR sensors are interfaced with Arduino Uno. 
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Motion System 

This Unit consists of four main parts; Motors, Motor Shield, Battery and Chassis. A Weight of robot is about 6Kg, 
four 300 RPM 12V DC motors of 3Kgcm torque have been used. The L293D motor shield is used to control the motors. 
The Motors are fitted to the legs of 1.5ft x 1ft Aluminium Chassis weighing around 1kg. The wheels are controlled over 
the Bluetooth using the android application. 

Communication 

The Built in Wi-Fi module is used to connect the Raspberry pi to a remote display. The robot is equipped with 
GSM module SIM900a for sending SMS to the default mobile number. 

Monitoring Unit 

The Raspberry Pi Camera module V2 is used for capturing the images and videos. Sony IMX219 8-megapixel 
sensor in v2 camera module enables it to capture high-resolution images and videos with low-light performance [7], [9]. 
The camera is mounted on the chassis at an angle of 45° to the ground. 

Human Detection Unit 

PIR sensors attached to the robot detect any human activity. A sensitivity of this passive infrared sensor is set to 
normal. Upon the perception of any motion in the area of detection by the sensor, Arduino board communicates with 
Raspberry Pi to initiate face detection. 

Power 

Motors and the GSM module are powered by 12V, 7.5Ah Lead-Acid Rechargeable battery whereas Arduino Uno 
and Raspberry Pi are powered by two separate batteries of 8F22 9V general purpose battery and 5V 5600mAh Li-ion 
battery respectively. 

Bluetooth Module 

Bluetooth module hc-05 is interfaced with Arduino Uno. The motion of the robot is controlled via the Android 
application using Bluetooth communication. Figure 2 demonstrates the surveillance robot which can be deployed in office 
premises for security purpose. 



(a) (b) 

Figure 2: Smart Office Surveillance Robot (a) Top View (b) Side View 
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SOFTWARE SPECIFICATION 

Programming Languages 

High-level python programming language is used for executing the facial detection algorithm, recognition 
algorithm and alert generation operations whereas object detection system is implemented in Arduino programming. 

Face Detection 

In this model, Haar Cascade Algorithm is used for executing face detection operation. Cascade files are created 
using this algorithm. Human face detection is executed using OpenCV tool. Detection of the object using the Haar feature 
based cascade classifiers is an excellent approach. The algorithm requires positive and negative images to train the 
classifier. Subsequently, features are extracted from it. The features are shown in the figures 3(a), 3(b) and 3(c) [2]. 



(a) Edge Features 


00 T^ine Features 



(c) Four-rectaiigle features 


Figure 3: Feature Extraction in Haar Cascade Algorithm 

The sum of the pixels in light rectangular boxes is subtracted from sum of the pixels in dark rectangular boxes 
representing a single value for each feature [1]. Features are computed quickly using integral imaging since only four 
pixels are used at once to define the image. The learning algorithm used is Adaboost. This algorithm has the capability of 
precisely choosing facial features from a large set. So, the efficiency of the classifier is eminent. The classifier discards the 
background region and hence helps in identifying regions with a better probability of finding the desired target [1]. 

Face Recognition 

Face recognition is a non-contact process so, it is advantageous over fingerprint or iris based methods [10]. The 
system capturing images does not intervene the activities of the person and clicks pictures maintaining some distance. 
Also, the identification process is independent and does not interrupt the person. In addition, crime rates can be lessened 
using face recognition technology as the face images captured can be stored in a database and retrieved to identify threats 
[ 11 ]. 

Real-time face recognition is done of the detected faces. The system identifies the person from a video frame. To 
recognize the face in a frame, first a face has to be detected in the frame. If it is present, a region of interest (ROI) is 
marked [3]. The ROI is processed for facial recognition. 

The entire process of face recognition can be divided into two parts: 

• Creating a database 

• Training 
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Creating Database 

This step involves taking pictures of the person for face recognition process. These images get stored in the 
database. For training, 20 pictures for every person are taken. Using the face detection algorithm, the region of interest is 
marked and the face is extracted, converted to grey scale and saved it to the database folder with different user IDs. 

Training 

The algorithm used here is Local Binary Patterns (LBP). Three methods of face recognition can be implemented 
using OpenCV:Eigenfaces, Fisherfaces and Local Binary Patterns LBP (Local Binary Pattern) [12]. 

Feature extraction is implemented using LBP algorithm. The image is divided into several regions. The LBP 
works with 8 pixels. Value of central pixel is considered to be the threshold for other neighboring 8 pixels. In case value of 
neighboring pixel is greater than or equal to the threshold value, then 1 is assigned to that pixel. Otherwise, 0 is assigned to 
it. Based on this LBP code is generated for that pixel. Similarly, for every pixel LBP code is calculated. The LBP for each 
pixel is calculated using equation 1 [5] 

LBP p r (x c y c ) = X III s(g P - g c )2 p (1) 

Where xc, yc are centre points of the pixel 
P is the no of neighbouring points 
g p is pixel value 
g c is threshold value. 

The feature vector of face image is obtained upon the calculation of LBP of each and every region of the face [5]. 
For finding similarities between the images the corresponding feature vectors are compared. 

Alert Generation 

A Short Message Service (SMS) is sent to the user in the case of an adverse situation. The user may not always be 
in a situation to attend a phone call, especially when the user is in a meeting. Hence alerting through an SMS is always 
better than a phone call. The robot is equipped with GSM module SIM900a for sending SMS to the default mobile number. 
As soon as a danger is detected, the GSM module is activated through I2C communication. The module uses an attention 
(AT) commands to send SMS to the user. 

Android Studio 

A Bluetooth application is made using Android studio as shown in Fig 4. Movement of the robot is controlled via 
this app. Bluetooth API(s) are used to make a connection with the Bluetooth module interfaced with Arduino Uno[8]. 
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Figure 4: Bluetooth Android App to Control the Motion of 
Robot using Commands: Left, Right, Forward, Backward and Stop 


RESULTS 



(a) (b) 

Figure 5: Face Detection 

(a)In Dim Light Condition (b) In Bright Light Condition 



Figure 6: Face Recognition as Known Person 
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Figure 7: Message Received from Raspberry Pi 3 Using GSM module 


The resolution of camera is 720 by 1080 pixels. According to Fig 5, upon detection of a human face, the face is 
encapsulated in a blue rectangular box. The size, color and shape of the rectangular box are coded in python language. The 
robot is tested for face detection in two conditions. One is in bright light and another in dim light. In both scenarios, face 
detection step gives a successful result. Also, in bright light condition, face detection gives a positive result when nearly 
half face is visible to the camera. 


Upon detection of a human face, the face recognition algorithm is triggered. According to Fig 6, the robot differentiates 
between a known person and an intruder by comparing the feature of face captured. Fig 7 illustrates the messages received 
upon discerning whether the human is a threat or not. In case of a known person, the user receives the name of the 
recognized person. However, in the case of a threat, message received is: “Message from Raspberry Pi. This is the 
unknown face.”. 


Table 1: Records Response Time for Various Time Critical Activities 


Time in seconds 

Activity 

Minimum 

Maximum 

Average 

Clicking Picture 

0.74 

0.94 

0.8343 

Face Detection 

1.95 

2.12 

2.0177 

Face Recognition 

0.12 

0.78 

0.3660 


Histogram for Face Detection 



Histogram for clicking picture 


“ 10 



Time to click picture (seconds) 


Figure 8 (a) 


Figure 8 (b) 
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Histogram for face recognition 



Face Recognition Time (seconds) 


Figure 8 (c) 

Table 1 summarizes the response time required for time-critical steps like capturing720 by 1080 resolution 
frames, face detection and face recognition. A total of 30 readings is taken to speculate the minimum and maximum time 
required for various steps. Thus, an average time required is calculated. Histograms are plotted based on the readings thus 
obtained. 

According to figure 8(a), the camera interfaced with Raspberry Pi clicks continuous images in a time range of 
minimum 0.751 to maximum 0.826 seconds. According to figure 8(b), 10 out of 30 samples require a minimum of 1.989 
seconds to a maximum of 2.032 seconds to execute face detection algorithm. Similarly, according to figure 8(c), 8 out of 
30 samples require a minimum of 0.313 seconds to a maximum of 0.408 seconds to execute face recognition algorithm. 

CONCLUSIONS / FUTURE WORK 

A surveillance robot has been developed for safeguarding a smart office. It detects human presence in its vicinity 
by employing the camera module and PIR sensors. The PIR sensors aid in sensing the motion of IR radiating objects. The 
human face is detected using camera module. Subsequently, when the robot detects a human face, it initiates the face 
recognition algorithm using Local Binary Pattern (LBP) approach. The robot sends an alert SMS to the user whenever it 
detects an unknown person in its vicinity. This robot demonstrates an effective way of safeguarding the office premises by 
providing an alert SMS on time when a threat is detected by it. The robot is unable to detect humans when the light is 
projected from behind the person since the area of the face becomes dark because of extreme light behind it. Future work 
has to be done in this regard. Also, an additional feature of tracing the human has to be implemented for increasing the 
efficiency of the robot. 
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